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ABSTRACT : 


The basic thesis of the larger paper from which ‘this 
condensation is drawn is that the use of evaluation contracts is 
advisable when evaluations are to be conducted by ‘persons external to 
the institution responsible for the program to be evaluated (or, in 
larger institution such as large universities, by persons external 
to the unit or department responsible for the program). The rationale 
presented in the later sections has led the authors to propose a set 
of criteria to assist administrators and evaluators as they think 
about whether to set up an evaluation contract and; if so, how to go 
about it. These criteria are summarized in this paper in the form of 
a checklist. The checklist contains seven sub-sections which deal 
respectively with thé following types of criteria: (1) Criteria for 
determining when to conduct an evaluation; (2) Criteria for 
determining whether to contract with an external ccntractor; (3) , 
Criteria to consider when selecting an evaluator; (4) Criteria for 
selecting among procedural options for letting a contract; (5) 
Criteria to consider when using a Request for Proposals; (6) Criteria + 
for use in negotiating the contract; and.(7) Criteria for use im f 
monitoring the contract. Different sections of the checklist will be 
useful to different- individuals and groups for different purposes. A 
flowchart, in which the major points of the checklist are translated 
into a pictorial sequence of events and decisions is also included. 
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- This paper is a condensation of a much larger document, "Standards and 

Procedures for Development and Implementation of an Evalyation Contraget" 
Prepared by the authors for the Alaska Department of Education) This sake 
shorter ‘version is intended for use by phose whg wish only an overview 
of the proposed standards and procedures sufficient to apply them. Those 
who would find it useful to study the extensive’ rationald ‘and discussion 
which supports the proposed standards and procedures or who may wish to 
peruse sample contraéts are referred fo the larger documents 
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. , _. Ty SUMMARY: AND OVERVIEW OF STANDARDS AND PROCEDURES 
p ‘ : FOR EVALUATION CONTRACTING 


‘ ‘ 9 “ ° : 
When a client and/or evaluator contemplate(s) entering into 4 contract 
to evaluate an edutational pen yray, or project, it can be presumed that 
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ae all prior decisions have been made. “4 me 
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pirets, ait evidently has bee dec'ided ehet an evathacion of the Progras 


; 4 w- 
s a or ptoject is appropriate. " Someqne senmuaty bane neeseee that there are is 


-Televang decisions to be made for which evaluative data ‘Would be Smyertant, 
‘or chet evaluative feedback ‘to program staff Gould eels: to ampEave, the 
program. Or perhaps there ix a "legal requtrement that the program be 


. 


evaluated. _Whatever the rationale for the decision, a0 intent to evaluate ' 
= ts patently a necessaty precursor to an an evaluation | contract. ~ | yo 
Sec6nd, ae. evidently has been decided’ that the siaihigtion should be - 
‘conducted by -an ind vidual or, agehcy external ay the educational institu | 
tion: esponsible for ane program or petact, Someone apparently’ has , 
‘ realized that certain eyese of evaluation poles toes ; end-of-project 
A edusidecs designed? to eel ‘funding agencies ig the school oo sky 
; accomplished what it seg out to accomplish, in the project) are ee 
suspece if conducted. by eeu personnel. Copcern for gredibilie) and oN : ‘ 
objectivity ‘of thd. evaluation results likely have prompted the decision * 
¥ to seek espera ee on Whatever the motivation, ‘contemplation | 
of an. seyatuagion contract i seed to m poaeee the notion of involvement 
DU gE am external patty, since only the most’ pessimistic are ‘Lisa to Fa 
(- require contracts when the evaluation is'to be conducted by indtviduale. 
- employed itn the same, institution. OE se 7 7 ae 
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Third, te evidently has been decided that the evaluation should b ; 


condueted under a contract of some similar formalized agreement, rather 


than having the external évaluator proceed to: see? one. the evaluation 


* 


. with guidance ae from whatever informal verbal or written Galogue 


may have octurred,. Someone apparently has decided that clear, written 

understandings of the. conditions. under. which the evaluation will be - 
: » e r2 . . - » 

¢artied out and of the expectations of all parties will help avoid later’ 


\ 


; + ‘ ; 25 
misunderstandings and disagreements concerning ghe conduct or results of 


° 


* the evaluation. , An. administrator who ‘has never yet received an evaluation’ .,:.., 


= “ ¥ {* . 


report in ‘time y for use for the decision for which it was intended a4 
- wish te require “delivery by’ a specified ‘date: as a ‘requirenent for fall 


payment for. the ‘evaluator’ 3 serviser “Or, an evaluator whose ee 
evaluation ‘report was rouriecegay the client: sesh removal of ali. ~« J 


negative findings) before presentation’ to aa funding agency way wish to 


a alae in a: formal contract that no changes be made in the evgtuation 
Ay . 

: —— i ‘ 

Thote without StpeeEt consent of. the authors, ° Perhaps. it ‘has been’ -% 

a A “. ; 


a concluded that, ee anong gdministrators. and evaluators with - the highed 


nonachia prefessional: standards and ethics, conflicts can and do arise 


“” t 


which could “have meen largely avoided jhad thete heed Setter documentation . 


at shared AUNGAT ERASE Logs goncerning important inten for couplatiig 
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the ‘evaluation, 


The basic thesis of the Lacgedp Pie from which this condensation 
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, is drawn is that the use of evaluation contracts a wdvisable shea 


evaluations are to be conducted by ‘persons external to the institution 
, pes : 5 
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responsible for the program to be evaluated (or; in larger institutions . 
: ‘ J od . s e: . ; , 7 
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such as large universities, by ‘persons external to the ea or department 


oe for: the prqgrany « The rationale presented in the Vacen sections 
I ig ied the authors tagpropose a set of criteria to assist administrators.-/ - 
‘and evaluators hie they think about Saackae to set up an evaluation contract’ 
and, af 50, how to go about it. These criteria are sumnatized-An. this 
paper in the form of a shecktter A flowchart, in which the major 


points of tle checklist are READ ELBE into a pictorial sequence of events 


and decisions is alsa included. ° ‘ 


Usets Guide to. the Checklist. 


Before prasenting the checklist, it may be MEARE to discuss briefly 
7 


its structure, ‘who might use’ it, when, and for what. 


First, the shepeiiap re seven sub-sections which deal respec- 


\ 
tively: with’ the following cypes, of epntects: 
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". Criteria for detavninise when tq Sonduct an evaluation 


o Os 
pee med for determining ‘Unethae to contract with an 
exter contractor fo ; % ‘ 


. ~ 


Criteria to “consider pen selecting an erelustne 
. Criteria for selecting among progedural options for + - 
» Letting a contract) map ts 38 yg. - 
* o . 
ia ‘Criteria to consider when uding a Request for BEVRCAESS 


“ge. 


6. Criteria for use ‘in ‘negotiating Jcheccontracp: 


7.° Criteria for usg in situa tase 
bal a 
Different sections of the checklist will be usefii’ to different | 
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individuals and groups for different’ purposes. Among the more obvious, 


> ¢ 


users would be those listed below. . - ee 


@ : Funding agency personnel (e.g., peysonnel in a State Department of 


‘ - of the checkhbist useful as they discharge their comprehensive responsi- 


. ‘Geo Ws . 
@ ow bilities. Section’ one could serve fo assist da decisions about which 


programs. would most profit from or require evaluation. Use of section 


. 


two should result in a decision. about whether evaluation should be 


e conducted internally or.externally,to the agency conducting the program. 


se _ 
Bosh Sections three through seven would be useful’ in cases Where funding 


«. é ‘ 
\ . . agency pérsonnel become involved’ in establishing the evaluation contracts 


¢ 


, 


@ . directly :as well as cdses where they, assist educational administrators to 


arrange ‘hase! iguanas under.titled funding programs (e.8., ESEA 


’ ¥ ‘ 
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Title I) but Late to the all el of the district to arrange. 


(eee Educational Sum snes LEseece (e.g., $chool superintendent or siiioival, 


Ss of a college of edutation)| would find all sections of the checklisg 
| 


useful in a manner parallel to that SSEREARES above for Aud ing apenty,. 
* pergonnel (with ‘the aceenelers of the proviso relating to tila ck 

: aducerionsl administrators) . 7 short, whénever a program or project is 
og in need of a ‘sugsat ive evaluation,’ the entire checklist should prove 

'.‘ useful to an admifistrator a ti ienedevint the best evaluatton 
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‘ ' ; help sueptbie to saitiy « out the ian of 
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- Educational. evaluators = have some ee in sections one, two,, 
ata on siaiek Avtgnence [tn section three 


; and four and would provide the 
would be made. However, active|participation, and use of the nabiiliog 


by thg evaluation contractor would involve se¢tions Yive shpoudh seven, \. 


e@ i 4 . j ‘ f . Pia os 


ae bear directly on planning}and corfducting the evaluation. , 
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Education Office. of Federal Programs) would likely find all seven sections. Pi 
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How to use the checklist. The checklist.is largely gelf-explanatory. 
Users would normAlly proceed through the relevant section(s) of the 
checklist, Shasta ng ein response option for each criterion listed. 

At the end of each section there is‘a summary question which should be 
answered on /the basis of sha eadoeuaaetn the payee questions in that. - 
‘saetie: tk ishould be noted that no~ formulae or rules are provided for 


how. one draws the sr | judgments fow each sectian.of the checklist. 


The checklist is proposed as an,aid to, and not # ORRRESUNES for, 
ae 
e 


professional jugggent. ‘In the authors" judgment, attempted rules: or 


algorithus fof summarizing criteria for each section would eenute in. a 


4 


mechanistic approach that’ would a “hay the ied daa fie which it-is 


intended.) _Each user must Becepe responsibility for his or her ‘own 
° a . 


‘judgment. This caveat RGCUSERECanAERS in cases where poveral persons 


7] 
apply/the checkfist to common entities, it may he REYOUASEE to agree/on 


- _common’ interp¥etations and weighting anne criteria. before aggregat ag: 


4 F 


individual judgments acrosé raterg. 
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Two other factors should also be noted. Rirst, it is not ‘assumed . 
; : ¥ 


pa 
eg ww , 23 = t ; 
that all users will have at their fingertips all the information necessary , 


to answer each fuaablon in the’checflist _(e. g-, questions dealing with |, 


the adequacy of “ha a abst ies es ln of a prospective evaluatén. 
a7 oat 
It is hoped that those who néed hs Sica will turn to others with 


knqwn ee in relevant areag eas questions require knowledge | 


bees that possessed by the user. If such help is ea the” 


-question-may simply have to gojtunanswered.. The risk of Aocluding some 


questions which cannot always /he answered a cenciucualiy seems less, 


~ 
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/ “However, than that of omitting the consideration from the checklist 
“Ye 5 
simply because obtaining an-answer ta that question is sometimes 


“difficult. 


\ Second, the several sections of the checklist deal with different 


“ considerations, making it inappropriate to’ reduce all sections to a 


. ‘ . 7 " ; 
completely consistent format \ In some sections, one item of ‘necessity 
leads to’ another in strict sequence, whereas other sections contain 

; profiles of items which could be easily answered ig d@fers other than 


‘that in’ which they are presented. These format changes’should not 


detract from the checklist if prospective ‘users remind themselves that 
_ the criteria:are intended to prompt thought about important aspects of 
: Si Z . -@ 4 


contracting for evaluations and facilitate decisions about that process 


and are not intended to produce a “scores”? © ww 
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|B. Checklist of Criteria for Evaluation Contracts - < a 
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The complete checklist is presented below, followed by the translation 


‘ . x e Z bog! ois. ‘ * 
of the checklist “into a flowchart of major events and decisions relating 
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to the criteria presented in the checklist. 
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CHECKLIST: Section Oné: Criteria for Determining + PAN 
‘ "When to Conduct an Evalvation Ke oe 


‘Checlt one for each item, | 


J 


Is there a legal: requirement to evaluate 
_ the program ? (If yes, initiate the 
\ evaluation; if-no,. go to Item 2 below. ) 


Ld . 
’ . . 


Is there a decision to be made for. 
which evaluation information would bé » 
relevant? (If yes, go to Item 3; if no, 
‘evaluation is inappropriate and you should 
~ discontinue further use of this checklist. ) 
.<) Fs 


& . 


v 


Is it likely that'thé evaluation will- provide 
dependable information relevant to the 
decision to be made? (If yes, go to Item 


4 if no, discontinue. ), 
~“ 


, Will the decision tie made Sactaniveld en . 
other bases andNininfluented by the ® oes 
evaluation data?” yes), evaluation is ee er a 
‘superfluous, aiecegeaay if‘no, initiate _ x4 
BVA Mat: ) 4 ANE rif oe ORE 


should an 
63, proceed 


CHNCKLIST, sacteg Two: “Criteria for Determinirg Whether to. * 
ane - Contract’ With an External Evaluator” ~ : * 
"Sk : 


ie : o Few . 33 _ , {Check one for each item’ 


. 


- : gh 3 “6 fave : \ 
~s there a legal réquirement that thé evaluation 
be conducted by an outside agency? (If yes, 
initiate the contract; if no, go to Item 2.) 


A re 2 We he Ananotad resources eee ‘to ‘capper’ 
contract for ‘an external evaluation ?. (If’ 


p /yes, proceed to Item 8: Tf no, conduct .the 
ra | evaluation internally and*#scontinue use 
of this checklist. ) wd 


ae 


e- 


ae rie 
Are credibility and objectivity of concern 
to the audiences for which the evaluatton is ~ 
conducted? (If yés, proceed to Item 4; if 
no, discontinue.) = < 


Is there an external contractor possessing 
the necessaty technical’competence wHo.is . 
available and willing to do the e luation?* 
(If yes, initiate the contract; if no, conduct 
the study internally, but recognize the loss 
of credibility and objectivity. ». 3 


e 
’ avatialar r? (if yes, =e - secfion three 
of the heey if no, discontinug its use. 


Further detail on this criterion will be a oviiiedl in/a subsequent section of this checklist. 


10 | 2, ‘ 
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(CHECKLIST: Section “Three: Criteta to Considér ‘When \ 


peeune an Evaluator ‘ : 
. ; Ss \ 


; f | ie 
(Each: item below is written to apply to an individual. If the potential contractor is an 
Agency the question should he re-cast acronyinety j : ; 


Gaia appears to be: 
‘(Check qne for each item 


Cannot . ’ Not. 
Well ‘Determine | Well jf 
| Qualified Qualifications .| Qualified} 


To what extent sigan the foomal diatinge of» «- 
the potential contractor qualify him/her to 
conduct evaluation studies? (Consider - 
‘ items such as major or minor degree 
, specialization; specific courses or.course 
. Séquences in Yaluation methodology; ' 
qualifications. of persons with whom 
training was taken, if such qualifications 
are. known. 
To what extent does the previous evaluation - 
experience of the potential contractor _«- 
qualify him/her to conduct evaluation 
studies? (Consider. items such a5 length | : 
of experience; relevance of experience. ) _ 
-To what extent does the previous performance 
of the potential evaluation contractor qualify 
him/her to conduct evaluation studies’ ? 


(As judged by work samples, or references), 
,_ce ~ 


SUMMARY TO SECTION "THEE: 


Based on questions: 1-3. above; to-whaf extent ae 
the potential evaluation contractor qualified to ¢ 
conduct e¥aluation studies? If "well qualified, " 
|proceed to’ subsequent sections of this checklist. 
If "not well qualified, e discontinue consideration 
of this catitractor. If "cannot determine qualifi-- 
cations": on basis of initial information, seek 
additionak information and/or assistance from 
colleagues. with expertise in évaluation to make 
this ‘critical judgment. . ‘ 


“NOTES A neutral rategory hds — in thie sect on of the checklist because informatior 
. initially available about potential cont rs may be insufficient eh permit unequivocal 


ae aia ee to.be made on this dimensié 


ae ede 
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CHECKLIST: Section Four: Criteria for Selecting Amon 


Ri “Rrocedural Options for Letting Contracts 


ie oe “Is there.a legal constraint against sole source 
’ contracting 2 (If yes, discontinue use of this 


section, of the checklist and go to Item 4 below; 


“ 


“RL Wg there: an eva naiee contractor who meets 
\. the: criteria in 'ehecklist three, who is ; 
\ intere sted: inthis evaluation, and whose *” 


negotiations. | 

Are the financial resources available re 
this evaluation ‘sufficient to warrant the cost 
‘of an RFP? (lf yes, prepare an REP; if no.’ 
usé synopsis review approach to find a . 
suitable con , lie 


' 


L] Sete Source Contract 


. be ; ‘ia Requey fgproposais 


[] Synopsis Review Approach ~ 


* evaluation studies a 
‘Proposals = = dontoaatel identified on the basis of a detailed proposal for the 
pos of the evaluation, cae ies in an open competition for the contract 


3 


, the evaluation study would be conducted and @) evaluator's hee 


. 


se , Las . rare 7 PSH 
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CHECKLIST: Section Five: Criteria to Consider When ' | 
Using an RFP E . 


| _ | Check one for each item 
Yes 


+ 4, Does the RFP include: 
e .. PY ls ° . | : : 
——tentntinnnnitnnnf...n adequate description of the context in pd re aan 
. which the evaluation is to take place? ~ if 
& ; a =e 
ei b, clear statements of the purposes and 
e characteristics of the project to be : 
Ro A, ae evaluated? . me 


¥ - 


c. the specific products expected from 
_the evaluation? 


e d. the resources available for the 
evaluation? 
- e, the constraints. which would affect the 
e ~ evaluation? 
; ‘ ,2. If methodological restrictions (e.g. , 
*- restrictions on sampling) are essential, 
_ are they included in the RFP? 
e 3. Has a panel been selected for reviewing — 
the proposals received? 
= 4: (Has the panel established criteria for 
tt ee” judging the proposals? 
e*- 
5. Are these criteria included in the RFP? 
* % e ? q 
@ 
Based on questions 1-5 above, -is the RFP 
? ready to be issued? (If yes, issue RFP and 
e proceed to section six of the checklist; if 
e@ me ® 


ll 


= Negotiating the Contract 


fF 
/ 
| 


are the negotiations scheduled to occur in 
time to permit evaluator involvement prior 
to the assignment of the treatment to the 
experimental units ? “ 
@ 2. If a time frame approach to the evaluation 
is appropriate, are the negotiations 
scheduled to occur in time to permit the 
collection of data (concerning the status 
: of those who are to participate) well in 
& advance of the introduction of the treatment? 
3: If necessary, are the negotiations scheduled 
to occur in time to permit the development, 
tryout, analysis, and revision of instruments 
prior to their first intended use? 
@ 4. Have adequate provisions been made for the 
~ development of an evaluation design through 
one of the following: preparation of design 
for set fee; preparation of design as the 
: . first phase of the contract; or preparation 
@-. of design under an initial contract, with a 
separate contract for implementing the design. 
5. Does the negotiated Scope of Work include: 


fT a. the procedures to be employed by the 
e evaluator and a time schedule for their 
* performance? 

b. adequate descriptions of the products to 
be expectedgrom the evaluation and 
deadlines for their delivery ? 

c. criteria for judging that the contract 
has been fulfilled? . 

* 6.- Have the negotiations dealt with the _ 
respective responsibilities of the client 
and contractor concerning: 

e ; a. the identification of the objectives of 
the inquiry? 

b. data access? 


ental design is appropriate, 


CHECKLIST: Section Six: Criteria for Use in 


™ 
= Check one for each item 


17. 


( 


‘Are the audiences of the evaluation. 
report(s) identified ? - 
Have negotiations concerning potential - 
conflicts of interest taken place? 

Has the issue of "piggybacking” other 


__ research interests been discussed ? 


Have the copyright and publication + 
rights issues been thoroughly 
explored and mutually agreed upon? 
Has.an agreement concerning the 
rights of prior review of reports been 
reached ? ; 

Has an adjudication procedure been 
established? 

Has a basis for payment been 
established ? 

Have the primary negotiators been 
identified ? 

Have those who will be aiiscted by the 
evaluation been invited to-.participate 
St a secondary level in the negotiations, 
at least by representation? 

Has a time and resource limit for 
negotiations been set by mutual agreement? 
If the negotiations have not been success- 
fully concluded with the time and resource 
limits set, has a second potential 
ontractor been identified? (Refer to 
checklist+section three. 


(lf yes, sign the contract; if no, return to section 
four and use to find another contractor. ) 


Check one for each item 


eo ; ; : a 
e ; ~b 
. "ao . 
CHECKLIST: Section Seven: Criteria for Use tn 
_ Monitoring the Contract. ‘ 


S : , ; |_Chéck one for each item 
tee ee MS 


i: Have procedures fom monitoring the progress of 
the work been agreed upon, i.e., have 
milestone review points and product deadlines 


=§ 


~y +e 
— * » 
2. _—_—«‘If circumstances are different than those 
6 anticipated in the contract, has a renegotiation 
occurred ? : . 


3. If renegotiation was not possible, has a mutually 
e * satisfactory agreement to terminate the contract 
been reached ? = 


4. If a dispute exists, have the two parties met to 
e attempt a resolution? Have the results of that 
meeting been documented ? 


If the dispute cannot be resolved among the | 
e@ parties to the contract, has the adjudication 
: ‘ process been invoked ? 


o 


* . = 


evaluation; if no, modify o 
necessary.) ¢ 


r terminate contract, * 


c. Flowctart of Events 
In tMe previous section of. this paper, the tomplete checklist 
relating to contract standards and procedures was presented. _ Lipche 


following pages, ‘that checklist has been transformed into a flowchart 


to-clarify—the-timing—aad sequenceof the events in the preparation 


of an evaluation contract. 


, ' The symbols use in this flowchart are as follows: 


Rectangle -- an event 


Diamond -- a decision point © 


ry - 


Oval --.a termination point for going through the flowchart 
« ? 


‘ 


Circle -- a continuation or°transfer point Raat S 


Hexagon -- end product desired E a 


7 ; : @ e e. 
e.. & e-. ee, -. @ = @ e e 7 


. 

. ‘Is there a 
legal requirement. 

to evaluate 


there a 
legal requirement 
‘that the evaluation be 
condycted by an | 
outside agency 


there a 
decision to-be 
made for which 
~eValuation information 
weuld be 
: relevant > 


Initiate 


evaluation 


_ Is 
it likely 
, that evaluation 
will provide dependable 
nformation relevant to 
that decision 


‘Are 
there 7 
financial regources 
available to support a 
contracted evaluatio 


“7 Will 
the decision 
be based on other 
factors, i.e., is the 
evaluation 
superfluous 


Initiate an 
internal evaluation 
but note loss of 
objectivity and 
credibility 


against sole source 
contracting 


tractor whorpossesses 
the necessary competencies 
“(use checklist 3) “and is 
willing to do thes 


Begin negotiations 
with identified 
contractor 


there another 
suitable contractor 
i.e., use checklis 
3 again) 
? 


financial © resources 
gufficient to warrant the 
cost of a request for 
proposals (RFP); 


Begin negotiations 
with this con- 
tractor — , 


Are ' E 
“all existing Sf ~ BN 
methodological : 
Does the RFP restrictions included REeeES y 
include: (a) An in the 
adequate description : ~ RFP 
of the context in, which + 
the evaluation is tq: take 
place? (b) Clear statements of 
the putposes ‘and characteristics of 
the project to be evaluated? (c) ‘The 
specific outcomes expected ‘from the evalua- 


. tion? (d) The o) "the pénatreinte for ‘ ' Hae 


evaluation? (e) The gonstraints which a panel aone.i 


selected for 
a x : reviewing the proposals 


would affect the e uation? : ; Z 


Select panel 


A : ve ‘Have : 
Z . criteria . 3 
Establish 
; established for a cet ‘i 


: a judging proposals |been Fl send ids artieria 
Rewrite _. |. ; included in eas 4 
me: * the RFP . : 


a 


Have 
the primary . . 
negotiators been _ Identify 
identified 


2. 


Have — 
those who ' 


will be 3ffected’ Identify and 
by the eva = 


solicit their 


nvited to participate at involvement 


° a secondary ‘level in 
the negotfations 
at least by 
-\represen- 
tatio 


~ Has . 
Ta-time and 
resource limit for 
negotiations been set by 


Select a . Mutual agreement 


contractor © 
_ and initiate 


f “tudes Fa OO ses aii ZH ak 
% < lion MAS 3 ear Ba 


ad ay P Se ee sa} 
2 A > 


, Regotiations 
scheduled to pro- 
* ceed in time-t6 permit . 
evaluation entry ‘at: the 

: ‘ appropriate point in time? 


‘If an éxperimental design. “1s ~afpropriate, 
- are the negotiations scheduled to occur in é 
f time to -permit evaluator - involvement prior : to’ the 
nies assignment of -the, treatment to she experimental ' units? 

If aL time frame’ approach to- the tie is appropriate,’ are, the: 
* negotiations scheduled ‘to occur in time to permit’ ‘the collection of 
data concérning the status of those who are to participa e well in agvance © 
“/ of the introduction of the treatment? s : 


‘Ife necessary, are the ‘Getettarions sctaduled to ,occur in time to, 
;Permit the development, tryout, analysis, and revision 
1 ¥ instruments prior: to here” first intended ‘uge?: 


. 


~~ 
Me 


: Can oN 
-negotia-— “a 
‘tions or - 
implementation 
‘schedules 
ibe revised 


atiquace.. 
provisions. been 


d (design ‘through one of the 
- following;—-preparation. of -a ieee 
i. design from a set fee; preparation; _ Has ‘ ; j j Sg, RS 
a mutt > A Se, : oes : ya te : 
- Yes’ agreeable destgn— Yes ‘7. ; vig 
been p ot 


established. 


ae” Can 
a Tne) design 

modification 

be negotiated 


Make adequate 
_ provisions for~ 
preparation of 
a,design .-- 


a Negotiate a ip ‘of: “boxe which ; 
includes: .. ~. Laan Aone 


* oe he 


Fy - oN ie 4 . 
. “thé procedures ‘to. be heirs 
3 ‘by tas and a time 
: Sachedule*for ‘thier performarice 
| b,; adequate-deseriptions of ‘the 
products. .tg be expected from 
Z the évaluation and deadTines.’ 
- for their délivety. < 


Ce! criteria’ for Judging that the : 


_ congratt: hab sheen fulfilled. 


; Negotiate each of the follauihe areas 
““g0 that the answer to each question 
»is "yes" * ‘the view of both parties: . 


SE 


Have~ “the dagectantuna! ‘deste with the 

respective responsibilities of the 

client and contractor toncerning? 

a. the identification of the objee~ . 
tives of the™ inquiry? | 


b. data access? P 
ee ee Se = at eoece Satis 


a mutiecaater the evaluation _ 
rt(s) “identified? _ 
Have negotiations Concerning potential 
conflicts of interest taken, place? 
Has the issue of "piggybacking" other 
research interests been discussed? 


‘Have the copyright and Webidentdoh 
rights issues been thoroughly explored 
and mutually agreed upon? : 

Hag an agreement. eoncerning the righte 
of prior review of ‘reports “been — 


‘reached? : . t 


[lenient iemeientcaniomieniantentententeny 


Has an ‘adjudication procedure been 


| established? 


—-———. eee in re re od re ene ee een ae gee eee” 


Has a basis of payment been © 


_ established? & 


en 


Establish. procedures 
| for monitoring the 
contract: 
a. procedural 
milestone 

_ Yreview points 
b. _product 
. deadlines 


’. 7 +Are : 
citcumstances - ‘ - Can 
, different than . : _y the contract 
-" those anticipated’ : be’ 
es in the~ : ~renegotiated 
‘contract. . ; : : 


Has 
a matually 
satisfactory 
agreement|to terminate 
the contract 
been 
reached 
2 


5 . contractor meet 

. - Renegotiate eo ge to resolve disputed 
and document~the 
results of that 
meeting 


Is the 
dispute 
resolved 
Successfully rie = Wee 
. concluded 
Evaluation 


ou 

Invoke the 

adjudication 
process — 


